Information processing apparatus, information processing method, and program

ABSTRACT

There is provided an information processing apparatus, an information processing method, and a program capable of looking back on the past two-way communication while maintaining connection between spaces in real time. The information processing apparatus controls two-way communication between a first space and a second space, and includes: a control unit that performs control to display first image information and second image information in the first space and performs control to display third image information and fourth image information in the second space, in which the first image information is current image information regarding the second space, the second image information is past image information regarding the second space, the third image information is current image information regarding the first space, and the fourth image information is past image information regarding the first space in the same time zone as that of the second image information.

FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

BACKGROUND

With the development of information processing technology in recentyears, various information processing systems that support an operationof various types of meetings such as a conference, a seminar, a lecturemeeting, or the like, performed in a company or an educational placehave been proposed.

For example, a remote conference system is an example of a conferencesupport system. In the remote conference system, display apparatuses,cameras, microphones, and speakers are installed in a plurality ofconference rooms located remotely, and a video and an audio acquired ineach conference room are transmitted to other conference rooms andoutput in real time, such that it becomes possible to simultaneouslyadvance a conference between the plurality of conference rooms locatedremotely.

Further, from the viewpoint of sharing of contents in real time by aplurality of persons who are at different places, for example, PatentDocument 1 discloses a content sharing system in which a plurality ofusers can view the same content in real time and also share a scroll ora mouse cursor with each other in real time. With such a system, it ispossible to realize communication in which the plurality of users chatabout a topical web page.

Further, Patent Document 2 discloses a content processing system capableof reproducing a state of a meeting together with an actually usedmaterial image by associating a moving image obtained by capturing thestate of the meeting and a presentation material image referred to inthe meeting with each other while synchronizing the moving image and thepresentation material image with each other on a time axis. With such asystem, a viewer can view a video and a material screen while comparingthe video and the material screen in the same time zone with each other.

CITATION LIST Patent Literature

Patent Literature 1: Japanese Patent No. 5857443

Patent Literature 2: JP 2005-260512 A

SUMMARY Technical Problem

However, the remote conference system described above or the contentsharing system described in Patent Document 1 is technology for sharinga phenomenon occurring in real time, such as viewing the same content inreal time or seeing states of each other's conference rooms, and did notshare the past phenomenon.

Further, in Patent Document 2, it is possible to view a video of thepast meeting and the material screen in the same time zone, butreproduction of two-way communication is not mentioned. In particular,it was not considered to reproduce the past two-way communication whilemaintaining current connection between a plurality of spaces.

Therefore, the present disclosure proposes an information processingapparatus, an information processing method, and a program capable oflooking back on the past two-way communication while maintainingconnection between spaces in real time.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing apparatus that controls two-way communication between a firstspace and a second space, including: a control unit that performscontrol to display first image information and second image informationin the first space and performs control to display third imageinformation and fourth image information in the second space, in whichthe first image information is current image information regarding thesecond space, the second image information is past image informationregarding the second space, the third image information is current imageinformation regarding the first space, and the fourth image informationis past image information regarding the first space in the same timezone as that of the second image information.

According to the present disclosure, there is provided an informationprocessing apparatus in which control of two-way communication between afirst space and a second space is performed, including: a control unitthat performs control to display first image information and secondimage information in the first space, in which the first imageinformation is current image information regarding the second space, andthe second image information is past image information regarding thesecond space and is image information of the same time zone as that ofpast image information regarding the first space displayed in the secondspace.

According to the present disclosure, there is provided an informationprocessing method that controls two-way communication between a firstspace and a second space, including: performing control to display firstimage information and second image information in the first space andperforming control to display third image information and fourth imageinformation in the second space, by a processor, in which the firstimage information is current image information regarding the secondspace, the second image information is past image information regardingthe second space, the third image information is current imageinformation regarding the first space, and the fourth image informationis past image information regarding the first space in the same timezone as that of the second image information.

According to the present disclosure, there is provided a program thatcauses a computer to function as an information processing apparatusthat controls two-way communication between a first space and a secondspace, in which the program causes the computer to further function as acontrol unit that performs control to display first image informationand second image information in the first space and performs control todisplay third image information and fourth image information in thesecond space, the first image information being current imageinformation regarding the second space, the second image informationbeing past image information regarding the second space, the third imageinformation being current image information regarding the first space,and the fourth image information being past image information regardingthe first space in the same time zone as that of the second imageinformation.

According to the present disclosure, there is provided an informationprocessing method in which control of two-way communication between afirst space and a second space is performed, including: performingcontrol to display first image information and second image informationin the first space, by a processor, in which the first image informationis current image information regarding the second space, and the secondimage information is past image information regarding the second spaceand is image information of the same time zone as that of past imageinformation regarding the first space displayed in the second space.

According to the present disclosure, there is provided a program thatcauses a computer to function as an information processing apparatus inwhich control of two-way communication between a first space and asecond space is performed and causes the computer to further function asa control unit that performs control to display first image informationand second image information in the first space, in which the firstimage information is current image information regarding the secondspace, and the second image information is past image informationregarding the second space and is image information of the same timezone as that of past image information regarding the first spacedisplayed in the second space.

Advantageous Effects of Invention

As described above, according to the embodiment of the presentdisclosure, it is possible to look back on the past two-waycommunication while maintaining connection between spaces in real time.

Note that the effect described above is not necessarily restrictive, andany effect set forth in the present specification or other effects thatcan be grasped from the present specification may be accomplishedtogether with or instead of the effect described above.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing an overview of an informationprocessing system according to an embodiment of the present disclosure.

FIG. 2 is a diagram for describing a state of communication using theinformation processing system according to the present embodiment.

FIG. 3 is a diagram illustrating an example of an entire configurationof the information processing system according to the presentembodiment.

FIG. 4 is a block diagram illustrating an example of a configuration ofan information processing apparatus according to the present embodiment.

FIG. 5 is a diagram illustrating an example of a menu selection screendisplayed in a virtual space according to the present embodiment.

FIG. 6 is a diagram illustrating an example of a method of selecting thepast video in a case where a recollection function according to thepresent embodiment is selected.

FIG. 7 is an example of each displaying the past videos that have beenseen from each other bidirectionally in each space using therecollection function according to the present embodiment.

FIG. 8 is an example of displaying both the past videos that have seenfrom each other in each space using the recollection function accordingto the present embodiment.

FIG. 9 is a diagram illustrating another example of a display size in acase of displaying both the past videos illustrated in FIG. 8.

FIG. 10 is a diagram illustrating another example of a display size in acase of displaying both the past videos illustrated in FIG. 8.

FIG. 11 is a diagram for describing a variation of a display method ofdisplaying the past videos in each space using the recollection functionaccording to the present embodiment.

FIG. 12 is a diagram for describing a variation of a display method ofdisplaying the past videos in each space using the recollection functionaccording to the present embodiment.

FIG. 13 is a diagram illustrating an example of displaying a silhouetteof a person extracted from the past video as a display example ofconveying an atmosphere of the past video using the recollectionfunction according to the present embodiment.

FIG. 14 is a diagram illustrating an example of arranging the personextracted from the past video using the recollection function accordingto the present embodiment with reference to depth information.

FIG. 15 is a diagram illustrating an example of synthesizing personseach extracted from the past videos of a counterpart side and an ownside using the recollection function according to the present embodimentwith each other on both sides.

FIG. 16 is a diagram illustrating an example of generating 3D models ofpersons each extracted from the past videos of the counterpart side andthe own side using the recollection function according to the presentembodiment and arranging the 3D models on virtual spaces.

FIG. 17 is a block diagram illustrating an example of a configuration ofa server according to the present embodiment.

FIG. 18 is a flowchart illustrating operation processing of theinformation processing apparatus in a case of executing the recollectionfunction according to the present embodiment.

FIG. 19 is a flowchart illustrating operation processing of the serverin a case of executing the recollection function according to thepresent embodiment.

FIG. 20 is a diagram illustrating another system configuration exampleaccording to the present embodiment.

FIG. 21 is a diagram illustrating an example of a configuration of aninformation processing apparatus according to a first embodiment ofanother system configuration example.

FIG. 22 is a flowchart illustrating operation processing of theinformation processing apparatus according to the first embodiment ofanother system configuration example.

FIG. 23 is a flowchart illustrating operation processing of theinformation processing apparatus according to the first embodiment ofanother system configuration example.

FIG. 24 is a diagram illustrating an example of a configuration of aninformation processing apparatus according to a second embodiment ofanother system configuration example.

FIG. 25 is a flowchart illustrating operation processing of theinformation processing apparatus according to the second embodiment ofanother system configuration example.

FIG. 26 is a flowchart illustrating operation processing of theinformation processing apparatus according to the second embodiment ofanother system configuration example.

FIG. 27 is a diagram illustrating an example of a configuration of aninformation processing apparatus according to a third embodiment ofanother system configuration example.

FIG. 28 is a flowchart illustrating operation processing of theinformation processing apparatus according to the third embodiment ofanother system configuration example.

FIG. 29 is a flowchart illustrating operation processing of theinformation processing apparatus according to the third embodiment ofanother system configuration example.

FIG. 30 is a diagram illustrating an example of a configuration of aninformation processing apparatus according to a fourth embodiment ofanother system configuration example.

FIG. 31 is a flowchart illustrating operation processing of a firstinformation processing apparatus according to the fourth embodiment ofanother system configuration example.

FIG. 32 is a flowchart illustrating operation processing of the firstinformation processing apparatus according to the fourth embodiment ofanother system configuration example.

FIG. 33 is a flowchart illustrating operation processing of a secondinformation processing apparatus according to the fourth embodiment ofanother system configuration example.

FIG. 34 is a flowchart illustrating operation processing of the secondinformation processing apparatus according to the fourth embodiment ofanother system configuration example.

FIG. 35 is a block diagram illustrating an example of a hardwareconfiguration of the information processing apparatus according to thepresent embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the accompanying drawings. Notethat, in the present specification and the drawings, components havingsubstantially the same functional configuration will be denoted by thesame reference numerals and an overlapping description will be omitted.

Further, a description will be given in the following order.

1. Overview of information processing system according to an embodimentof the present disclosure

2. Configuration

2-1. Configuration of information processing apparatus 10

2-2. Configuration of server 20

3. Operation processing

3-1. Operation processing of information processing apparatus 10Aregarding execution of recollection function

3-2. Operation processing of server 20 regarding execution ofrecollection function

4. Another system configuration example

4-1. First embodiment

4-2. Second embodiment

4-3. Third embodiment

4-4. Fourth embodiment

5. Hardware configuration

6. Summary

1. OVERVIEW OF INFORMATION PROCESSING SYSTEM ACCORDING TO AN EMBODIMENTOF THE PRESENT DISCLOSURE

FIG. 1 is a diagram for describing an overview of an informationprocessing system according to an embodiment of the present disclosure.As illustrated in FIG. 1, an information processing system 1 accordingto the present embodiment is a space adjacent telepresence system thatconnects spaces (for example, space A and space B) located at distantplaces to each other by a video, an audio, or other informationchannels.

Specifically, for example, when space information (videos and audios) isobtained by cameras and microphones (not illustrated) installed,respectively, in space A and space B, the space information istransmitted to a space of a counterpart side in real time, and is outputfrom a display 131 and a speaker (not illustrated) installed on, forexample, a wall of a living room, a dining room or the like of thecounterpart space. The display 131 is a virtual window (or a door or anentrance) connected to the counterpart space and a state of the space ofthe counterpart side can be seen from the virtual window. Therefore, auser can feel as if the space of the counterpart side is the immediatelynext room to feel as if he/she is at the same place as a place where acounterpart user is. Here, a state of communication using theinformation processing system according to the present embodiment willbe described with reference to FIG. 2.

As illustrated in FIG. 2, for example, in space A, a large display 131Ais installed on a wall or the like of a room. A camera, a microphone,and a speaker are installed around the display 131A or on the display131A. A plurality of cameras, a plurality of microphones, and aplurality of speakers may also be provided. Further, arrangement placesof the camera, the microphone, and the speaker are not particularlylimited, but it is preferable that the camera captures an image of astate of space A using a direction from the display 131A toward aninterior of the room as an image capturing direction. Note that, inspace B, a display, a camera, a microphone, and a speaker are similarlyinstalled.

On the display 131A, a video acquired in space B to which space A isconnected is displayed in real time. On the display 131A, a video 310 ofspace B may be displayed on the entire screen (in this case, a userfeels as if space B exists immediately next to space A through a wall ofspace A), or as illustrated in FIG. 2, a video 310 of space B may bedisplayed at a slightly distant place by giving a depth to the video 310and a virtual space may be generated between space A and space B. In acase where the virtual space is generated, for example, a passage image320 may be displayed so that the video 310 of space B and a virtualwindow (a frame of the display 131A) of space A are seen adjacent toeach other.

Further, a size and an installation position of the display 131A are notparticularly limited. For example, as illustrated in FIG. 2, the display131A may be larger than a human and a lower side of the display 131A maybe located near a floor, or the display 131A may have about a size ofthe upper body of a human and a lower side of the display 131A may belocated at about a height of the waist of the human.

Further, the camera, the microphone, and the speaker are installedaround the display 131A. Since an audio (that is not limited to anuttered voice, but is various environmental sounds including a livingsound such as a sound of door opening and closing or a television, afootstep sound, and the like) acquired in the counterpart space isoutput from the speaker (not illustrated), even though a figure of acounterpart is not visible, the user can feel an atmosphere or a sign ofthe counterpart space. Particularly, in a case of always connectingrooms of family living apart from each other to each other, each other'ssituations can be confirmed at any time, and anxiety that does not knoweach other's states in remote places is thus reduced, such that a senseof security can be obtained.

Further, as a conventional example, in a video chat using an informationcommunication terminal such as a smartphone, a personal computer (PC) orthe like, a conversation with a purpose has been made by calling thecounterpart when there is a thing to do, but in the space adjacenttelepresence system according to the present embodiment, a space isbasically always connected to the counterpart space, and it is thus easyto feel free to start to talk or casually make a conversation, such thatan effect increasing communication occurs. Further, since a voice of thecounterpart is heard even in a state where a face of the counterpart isnot seen from the virtual window, it is possible to make a conversationwithout seeing the face.

BACKGROUND

Here, in the space adjacent telepresence system, it is possible toeasily perform communication by connecting the space to the counterpartspace in real time, but it is also conceivable to later recollect adaily state or a special event that has occurred in that place andperforms communication such as looking back on the past with aconnection counterpart.

Therefore, in the present embodiment, in the space adjacent telepresencesystem, it becomes possible to look back on a state of the past event(that is, the past two-way communication) that has occurred in thatspace while maintaining an effect that it seems that the spaces areadjacent to each other in real time.

Specifically, by storing space information such as videos, audios andthe like of each space, synchronizing states (past videos) ofcounterpart spaces that have been seen from virtual windows of eachother's spaces with each other on a time axis, and playing back thestates (past videos) in each space, it is possible to look back on thepast states (past two-way communication) of the counterpart spaces thathave been seen in the respective spaces at the same timing. Such afunction of looking back on the past state is provided as a“recollection function” in the space adjacent telepresence systemaccording to the present embodiment.

Such an information processing system according to the presentembodiment has, for example, a configuration as illustrated in FIG. 3.FIG. 3 is a diagram illustrating an example of an entire configurationof the information processing system according to the presentembodiment.

As illustrated in FIG. 3, the information processing system 1 accordingto the present embodiment includes a plurality of information processingapparatuses 10 (10A and 10B) and a server 20. The information processingapparatus 10A, the information processing apparatus 10B, and the server20 are connected to each other through a network 30.

Each information processing apparatus 10 controls input and output ofspace information using an input unit and an output unit provided ineach space. Specifically, the information processing apparatus 10Aacquires space information such as a video or an audio of space A froman input unit installed in space A, and transmits the acquired spaceinformation to the server 20 or the information processing apparatus10B. Further, the information processing apparatus 10A performs controlto output space information (a video and an audio) of the informationprocessing apparatus 10B received from the information processingapparatus 10B or the server 20 from an output unit provided in space A.

The server 20 stores the space information of each space transmittedfrom each information processing apparatus 10. Further, in a case wherelook-back on the past of the two-way communication is performed in thespace adjacent telepresence system, the server 20 performs control toappropriately transmit the accumulated past space information to theinformation processing apparatus 10A and the information processingapparatus 10B or control to synchronize the past space information ofeach space with each other on the time axis.

The information processing system according to the embodiment of thepresent disclosure has been described hereinabove. Next, a specificconfiguration of each apparatus included in the information processingsystem according to the present embodiment will be described withreference to the drawings.

2. CONFIGURATION

<2-1. Configuration of Information Processing Apparatus 10>

FIG. 4 is a block diagram illustrating an example of a configuration ofan information processing apparatus 10 according to the presentembodiment. As illustrated in FIG. 4, the information processingapparatus 10 includes a control unit 100, an input unit 110, anoperation interface (I/F) 120, an output unit 130, a communication unit140, and a storage unit 150.

(Control Unit 100)

The control unit 100 functions as an arithmetic processing device and acontrol device, and generally controls an operation in the informationprocessing apparatus 10 according to various programs. The control unit100 is realized by, for example, an electronic circuit such as a centralprocessing unit (CPU), a microprocessor or the like. Further, thecontrol unit 100 may include a read only memory (ROM) storing programs,operation parameters or the like that are used and a random accessmemory (RAM) temporarily storing parameters or the like that areappropriately changed.

The control unit 100 performs control to acquire various spaceinformation from the input unit 110, accumulate the acquired spaceinformation in the storage unit 150, and transmit the acquired spaceinformation from a transmitting unit 141 to the server 20. From theinput unit 110, for example, a video, an audio, depth information, andthe like are obtained. The control unit 100 may allow information onwhich predetermined processing such as noise processing or compression,or the like, has been performed to be accumulated and transmitted.

Further, the control unit 100 can also perform user recognition on thebasis of the various space information obtained from the input unit 110.As the user recognition, for example, user identification to identifywho a user is by comparison with a previously registered face image orsituation recognition such as a position (where the user is in a room),a posture (whether the user are walking, sitting, standing, or thelike), emotion (whether the user is laughing, angry, or the like),behavior (what the user is doing), or the like, of the user is assumed.

Further, the control unit 100 can recognize an uttered voice of the useron the basis of audio information obtained from the input unit 110.

Further, the control unit 100 can recognize an environment (for example,object recognition (a position and a type of object) of the space on thebasis of the various space information obtained from the input unit 110.

The control unit 100 may accumulate such various recognition results inthe storage unit 150 or transmit such various recognition results fromthe transmitting unit 141 to the server 20 in a state of including suchvarious recognition results in the space information.

Further, the control unit 100 according to the present embodiment alsofunctions as a function selecting unit 101, an image synthesizing unit102, and an output control unit 103.

The function selecting unit 101 selects a function that can be used inthe space adjacent telepresence system according to the presentembodiment. For example, the function selecting unit 101 acquiresinformation of a user operation performed using a remote controller orthe like, from the operation I/F 120, and selects a function accordingto the user operation. As available functions, for example, an albumfunction of viewing a captured image captured by a digital camera or thelike together with a counterpart user, a game function of playing acertain game with the counterpart user using a virtual space, arecollection function (function of looking back on the state of the pastaccording to the present embodiment) of looking back on two-waycommunication with a counterpart space with the counterpart user, andthe like, are assumed. Here, a method of selecting a menu will bedescribed with reference to FIG. 5.

As illustrated in FIG. 5, an image 300 including the video 310 of thealways connected counterpart space and the passage image 320 of avirtual space connecting the counterpart space and a user side to eachother is displayed on the display 131. At this time, according to apredetermined trigger operation (for example, a gesture, a voice, atouch operation, a button operation, or an operation by a remotecontroller of the user), a menu icon 330 is displayed on the virtualspace and in front of the video 310 of the counterpart space, asillustrated in FIG. 5. An image of the icon illustrated in FIG. 5 is anexample, and the present embodiment is not limited thereto. For example,an icon 330 a is an icon for the album function, an icon 330 b is anicon for the game function, an icon 330 c is an icon for therecollection function, an icon 330 d is an icon for a tool function, andan icon 330 e is an icon for a setting function. The user selects theicon by a remote controller (not illustrated), a gesture, a voice, atouch operation on a screen of the display 131, an operation of anoperation unit (not illustrated) such as a button or the like providedaround the screen, or the like.

Here, an example of a method of selecting the past video in a casewhere, for example, the icon 330 c for the recollection function isselected will be described with reference to FIG. 6. In a case where theicon 330 c for the recollection function is selected, a calendar image331 is displayed, for example, as illustrated on the left side of FIG.6. When a date to perform look-back is selected from the calendar image331, a seek bar 332 for searching the past video of the selected date isdisplayed as illustrated on the right side of FIG. 6. The user operatesthe seek bar 332 to select a time zone to perform look-back whileconfirming the past video 333 displayed correspondingly. The past video333 to be searched may be a video of the counterpart space that has beenseen from a space of an own side or may be a video of a space of an ownside that has been seen from the space of the counterpart side. Notethat the method of searching and selecting the past video illustrated inFIG. 6 is an example, the present embodiment is not limited thereto, andthe method of searching and selecting the past video may be anothermethod as long as a date or a time can be designated.

When the past video of the time zone selected by the recollectionfunction is transmitted from the server 20, the image synthesizing unit102 performs processing for synthesizing the past video with a real timevideo of the currently connected counterpart space. From the server 20,in addition to the past video of the counterpart space, a processedcontent (for example, an image or a 3D model in which only a person isextracted from the past video) generated on the basis of the past video,or a synchronized content in which the past video of the counterpartspace and the past video of the space of the own side are synchronizedwith each other on the time axis can be received.

The output control unit 103 performs control to output real-time spaceinformation of the always connected counterpart space, received by areceiving unit 142, from the output unit 130. For example, the outputcontrol unit 103 performs control to display a real-time video capturedin space B on the display 131 provided in space A and output a real-timeaudio collected in space B from the speaker (not illustrated) providedin space A. The display 131 is a large display unit installed on thewall or the like or the like of the room, for example, as illustrated inFIG. 2, and the video of the counterpart space displayed on such adisplay 131 is a video captured at an appropriate angle of view so as tobe seen without any discomfort in a case where the display 131 is thevirtual window. Such an arrangement of the display 131 and such displayof the video of the counterpart space by the display 131 are similarlyperformed in the counterpart space. Therefore, it seems that eachother's spaces are adjacent to each other, such that each user canperform communication bidirectionally through the display 131.

The output control unit 103 may display the video of the counterpartspace in an expression arranged in the virtual space (that is, displaythe video of the counterpart space with a distance from an adjacentspace) or display the video of the counterpart space over the entirescreen of the display 131 without providing the virtual space (that is,without a distance from an adjacent space), on the display 131. Varioussetting contents such as which display method to select, or how far tobe distant from the virtual space or how to set a scene in the virtualspace in a case where the virtual space is provided, and the like, maybe set automatically (by default) or may be selected by the user.Further, such a setting may be shared with a display method in thecounterpart space.

Further, the output control unit 103 can perform control for displayingthe image synthesized by the image synthesizing unit 102 described aboveon the display 131. Here, FIGS. 7 to 16 illustrate specific examples ofsynthesis of the past videos and the like when the recollection functionis used. Which display method is used may be set automatically (bydefault) by a system side or may be arbitrarily selected and changed bythe user. Further, a display method selected by any user can be sharedbetween space A and space B.

FIG. 7 is an example of each displaying the past videos that have beenseen from each other bidirectionally in each space. In this case, asillustrated in FIG. 7, on a display 131B of space B where user B is, thepast video 333 of space A that has been seen from space B is arranged ona virtual space in front of a current real-time video 340 of space B. Onthe other hand, on a display 131A of space A where user A is, the pastvideo 335 of space B that has been seen from space A is arranged on avirtual space in front of a current real-time video 310 of space B.These past videos 333 and 335 are the past videos of the same time zone,and are played simultaneously in each space. Therefore, each user canlook back on the past two-way communication while maintaining connectionbetween the spaces in real time (that is, while conversing with a userof the counterpart space in real time). For example, user A is the sonof user B, and can show a state of a birthday party of a grandson (userC) performed in space A to user B who was not at that place. Here, thetwo-way past videos are displayed, respectively, but the past audioscollected at the same time can also be output.

FIG. 8 is an example of displaying both the past videos that have beenseen from each other in each space. In this case, as illustrated in FIG.8, on the display 131B of space B where user B is, the past video 333 bof space A that has been seen from space B and the past video 335 b ofspace B that has been seen from space A are arranged on the virtualspace in front of the current real-time video 340 of space B. The pastvideo 333 b and the past video 335 b simultaneously displayed on thedisplay 131B are the past videos of the same time zone, and aresynchronized contents that have been synchronized with each other on thetime axis by the server 20. On the other hand, on the display 131A ofspace A where user A is, the past video 335 a of space B that has beenseen from space A and the past video 333 a of space A that has been seenfrom space B are arranged on the virtual space in front of the currentreal-time video 310 of space B. The past video 333 a and the past video335 a simultaneously displayed on the display 131A also are the pastvideos of the same time zone, and are synchronized contents that havebeen synchronized with each other on the time axis by the server 20.Each synchronized content is played simultaneously in each space.Therefore, each user can look back on the videos of both sides that havebeen seen from each other in the past two-way communication whilemaintaining connection between the spaces in real time (that is, whileconversing with a user of the counterpart space in real time). In thiscase, for example, user B can see not only the past state of thecounterpart space (space A), but also the past state of the space (spaceB) of the own side in the same time zone, and can thus simultaneouslysee a state of user D who is in space B, seeing a state of space A inwhich the grandson (user C) opens a present and plays, for example.Further, here, naturally, not only the display of the two-way pastvideos but also the past audios collected at the same time can beoutput.

In the example illustrated in FIG. 8, sizes or arrangements of thedisplay of the past videos can be appropriately changed by a useroperation or the like. For example, in the example illustrated in FIG.8, the past video of the counterpart space is arranged in front of thepast video of the space of the own side on the virtual space, but thepast video of the counterpart space and the past video of the space ofthe own side may be moved forward and backward on the virtual spacewhile maintaining a positional relationship therebetween. Further, abalance of the display size may be changed as illustrated in FIG. 9 orFIG. 10 while maintaining a front and rear positional relationship.

Further, as a variation of a method of displaying the past video on thevirtual space, in addition to arranging the past video simply on thevirtual space as illustrated in FIGS. 7 to 10, the past video may beexpressed so as to be projected upward from an apparatus placed on thepassage image 320 of the virtual space, as illustrated in FIG. 11, ormay be expressed so as to be put in a photo frame placed on the passageimage 320 of the virtual space, as illustrated in FIG. 12.

FIG. 13 is a diagram illustrating an example of displaying a silhouetteof a person extracted from the past video as a display example forconveying an atmosphere of the past video. In this case, as illustratedin FIG. 13, on the display 131B of space B where user B is, a silhouetteimage (for example, a silhouette of user C, a moving image) of a personextracted from the past video 333 of space A that has been seen fromspace B is synthesized with the current real-time video 340 of space B.Since a background of the space is the same, a movement or an atmosphereof a person who was in that space in the past can be sensed only bysuperimposing and displaying a silhouette of the past person on acurrent video. On the other hand, on the display 131A of space A whereuser A is, a silhouette image (for example, a silhouette of user D, amoving image) of a person extracted from the past video 335 of space Bthat has been seen from space A is synthesized with the currentreal-time video 310 of space B. These silhouette images are the pastvideos of the same time zone, and are played simultaneously in eachspace.

An expression of the person extracted from a past image may be asilhouette (shadow) as illustrated in FIG. 13, may be a blurredexpression, or may be a bone expression (an expression of a stick figuregenerated by recognizing a skeleton of the body). Further, the extractedperson may be displayed as it is or may be displayed transparently.

Further, a position where the image of the extracted person issynthesized may be arranged on the basis of which position he/she was inthe room, for example, on the basis of depth information.

Further, the image of the extracted person may be displayed in front ofa person appearing in a current video, or may be displayed in front ofthe person appearing in the current video in a case where the extractedperson is located in front of the person appearing in the current videoand be displayed so as to move to the rear of the person appearing inthe current video in a case where the extracted person is located behindthe person appearing in the current video, on the basis of the depthinformation. An example is illustrated in FIG. 14. In the exampleillustrated in FIG. 14, users C and E extracted from the past video ofspace A are appropriately arranged in the same background of space A onthe basis of depth information of the past video. Therefore, it can beunderstood that in the past video, user C was located in front of aplace where user A is currently located and user E was located behinduser A.

Note that only the person extracted from the past video of thecounterpart space has been synthesized in FIGS. 13 and 14, but thepresent embodiment is not limited thereto, and a person extracted fromthe past video of the space of the own side may be further synthesized.An example is illustrated in FIG. 15. In this case, as illustrated inFIG. 15, on the display 131B of space B where user B is, the silhouetteimage (for example, the silhouette of user C, the moving image) of theperson extracted from the past video 333 of space A that has been seenfrom space B and the silhouette image (for example, the silhouette ofuser D, the moving image) of the person extracted from the past video335 of space B that has been seen from space A are synthesized with thecurrent real-time video 340 of space B. On the other hand, on thedisplay 131A of space A where user A is, the silhouette image (forexample, the silhouette of user D, the moving image) of the personextracted from the past video 335 of space B that has been seen fromspace A and the silhouette image (for example, the silhouette of user C,the moving image) of the person extracted from the past video 333 ofspace A that has been seen from space B are synthesized with the currentreal-time video 310 of space B. These silhouette images are the pastvideos of the same time zone, and are played simultaneously in eachspace. Further, the image of the person extracted from the past video ofthe space of the own side is arranged at the foremost side of thevirtual space, as illustrated in FIG. 15, and moreover, may be asilhouette inverted in a horizontal direction in order to make the userperceive that the user is looking the person from behind. Alternatively,a back view of the person generated on the basis of a 3D model may besynthesized. Further, all the extracted persons have been displayed assilhouettes in the example illustrated in FIG. 15, but the presentembodiment is not limited thereto. Display methods of the extractedpersons may be different from each other, such as, for example,displaying a person of a front side (the person extracted from the pastvideo of the space of the own side) as a silhouette and transparentlydisplaying the person extracted from the past video of the counterpartspace as illustrated in FIG. 14, or the like.

FIG. 16 is a diagram illustrating an example of generating 3D models ofpersons each extracted on the basis of the past videos of both spacesand arranging the 3D models so as to face each other through windowframes on the virtual spaces. In this case, as illustrated in FIG. 16,on the display 131B of space B where user B is, a 3D image (for example,a 3D image of user C, a moving image) of the person extracted from thepast video 333 of space A that has been seen from space B and a 3D image(for example, a 3D image of user D, a moving image) of the personextracted from the past video 335 of space B that has been seen fromspace A are arranged on a passage image 320 b of the virtual spacelocated in front of the current real-time video 340 of space B. In thepassage image 320 b of the virtual space, the 3D images of each personare arranged to face each other across virtual windows, and arelationship through windows expressing adjacent spaces is expressed.Further, positions of the 3D images of each person from the windowframes may be appropriately arranged on the basis of depth information.On the other hand, also on the display 131A of space A where user A is,similarly, the silhouette image (for example, the silhouette of user D,the moving image) of the person extracted from the past video 335 ofspace B that has been seen from space A and the silhouette image (forexample, the silhouette of user C, the moving image) of the personextracted from the past video 333 of space A that has been seen fromspace B are synthesized with the current real-time video 310 of space A.The 3D images of these extracted persons are generated on the basis ofthe past videos of the same time zone, and are played simultaneously ineach space.

As such, it is possible to reproduce the past two-way communication onthe virtual space while maintaining adjacency with the counterpart spaceto look back on states of both sides together with the counterpart user.In 3D modeling, not only persons but also states around the persons maybe modeled. Further, a size of the entire 3D model arranged in thevirtual space can be appropriately changed, and may be changed by, forexample, a user using a remote controller or the like.

(Input Unit 110)

The input unit 110 has a function of inputting the space information tothe control unit 100. For example, the input unit 110 is realized by acamera, a microphone, and a sensor. The number of cameras may be plural,and the cameras capture an image in a space (for example, a living room)to acquire a captured image. Further, the number of microphones may beplural, and the microphones collect sounds in a space (for example,sounds in the room, sounds that are heard from the outside of the room,or the like) to acquires audio data. Further, the sensor has a functionof detecting various information in the space, and for example, a depthsensor (distance sensor), a human sensor, an infrared sensor, anilluminance sensor, and the like are assumed.

(Operation I/F 120)

The operation I/F 120 receives operation input by the user, and outputsoperation information to the control unit 100. The operation I/F 120 isrealized by, for example, a button, a switch, a keyboard, a touchsensor, and the like, and is provided around the display 131, or thelike. Further, the operation I/F 120 is connected to a remote controlleroperated by the user by wireless/wired communication, and can obtainoperation input information by the user. The remote controller may be aninfrared (IR) remote controller, or may be a Bluetooth (registeredtrademark) remote controller. Further, the remote controller may be adedicated terminal, may be realized by an information communicationterminal such as a smartphone, a tablet terminal or the like, or may besubstituted by a game device (a game pad or the like). Further, input ofthe user operation can also be made by a gesture such as a movement of ahand, or an uttered voice. The analysis of the gesture or the utteredvoice can be performed by the control unit 100 on the basis of thecaptured image and the audio information obtained from the input unit110.

(Communication Unit 140)

The communication unit 140 is connected to the network 30 in a wired orwireless manner, is connected to the server 20 or another informationprocessing apparatus 10 through the network 30, and performstransmission and reception of data. The communication unit 140 iscommunicably connected to the network 30 by, for example, awired/wireless local area network (LAN), Wi-Fi (registered trademark),long term evolution (LTE), third generation mobile communication system(3G), or the like. Further, as illustrated in FIG. 4, the communicationunit 140 includes the transmitting unit 141 that performs transmissionof information and the receiving unit 142 that performs reception ofinformation, as functional configurations.

(Output Unit 130)

The output unit 130 outputs, for example, real-time space information ofthe counterpart space according to the control of the control unit 100.For example, the output unit 130 includes the display 131 and thespeaker, and performs output of an image and output of an audio. Here,the display 131 is used as an example, but an image display apparatus isnot limited to the display 131, and may be, for example, a projector.The information processing apparatus 10 may project a real-time video ofthe counterpart space on the wall of the room.

(Storage Unit 150)

The storage unit 150 is realized by a Read Only Memory (ROM) storingprograms, operation parameters or the like, that are used for processingof the control unit 100 and a Random Access Memory (RAM) temporarilystoring parameters or the like that are appropriately changed.

The configuration of the information processing apparatus 10 accordingto the present embodiment has been specifically described hereinabove.Note that a configuration of the information processing apparatus 10 isnot limited to the example illustrated in FIG. 4, and for example, someof configurations may be realized by a separate body and may beconnected to the information processing apparatus 10 in a wireless/wiredmanner. That is, the information processing apparatus 10 may beconfigured by a plurality of apparatuses. Further, some of processingperformed by the control unit 100 of the information processingapparatus 10 may be performed by the server 20 or an intermediate server(not illustrated) located between the information processing apparatus10 and the server 20. Therefore, a processing load of the informationprocessing apparatus 10 is reduced.

<2-2. Configuration of Server 20>

FIG. 17 is a block diagram illustrating an example of a configuration ofthe server (information processing apparatus) 20 according to thepresent embodiment. As illustrated in FIG. 17, the server 20 includes acontrol unit 200, a communication unit 210, and a content storage unit220.

(Control Unit 200)

The control unit 200 functions as an arithmetic processing device and acontrol device, and generally controls an operation in the server 2according to various programs. The control unit 200 is realized by, forexample, an electronic circuit such as a Central Processing Unit (CPU),a microprocessor or the like. Further, the control unit 200 may includea ROM storing programs, operation parameters or the like that are usedand a RAM temporarily storing parameters or the like that areappropriately changed.

The control unit 200 performs control to accumulate the spaceinformation transmitted from each information processing apparatus 10 inthe content storage unit 220. Further, the control unit 200appropriately transmits necessary information to the informationprocessing apparatus 10 or performs output control in the informationprocessing apparatus 10 according to a request from the informationprocessing apparatus 10.

For example, in a case where there is a request to execute therecollection function from the information processing apparatus 10, thecontrol unit 200 extracts (retrieves) a content (that is, the past videoand the past audio) of a designated date and time from the contentstorage unit 220 and transmits the content to the information processingapparatus 10, and performs control to display the content together withthe current video of the counterpart space on the display 131.

Further, the control unit 200 according to the present embodiment alsofunctions as a content processing unit 201 and a content synchronizingunit 202. The control unit 200 transmits a processed content obtained byprocessing the past video by the content processing unit 201 or asynchronized content obtained by synchronizing the past videos (pastvideos of space A and space B) of both sides with each other by thecontent synchronizing unit 202 to the information processing apparatus10 and performs control to display the processed content or thesynchronized content with the current video of the counterpart space onthe display 131, according to a request from the information processingapparatus 10.

The content processing unit 201 generates a person image by extractingonly a person from a predetermined past video or generates a 3D model ofa person appearing in the past video from three-dimensional informationbased on the past video and corresponding sensor information (sensorinformation of the same time zone), in the use of the recollectionfunction.

The content synchronizing unit 202 generates a content (also referred toas a synchronized content) in which the selected past videos of space Aand space B of the same time zone are associated (synchronized) witheach other on the same time axis, in a case of displaying the pastvideos of both sides in each space as described with reference to FIG.8.

(Communication Unit 210)

The communication unit 210 is connected to the network 30 in a wire orwireless manner, and transmits and receives data to and from eachinformation processing apparatus 10 through the network 30. Thecommunication unit 210 is communicably connected to the network 30 by,for example, a wired/wireless Local Area Network (LAN), wirelessfidelity (Wi-Fi) (registered trademark), or the like.

(Content Storage Unit 220)

The content storage unit 220 stores the space information transmittedfrom each information processing apparatus 10. Further, the contentstorage unit 220 may store the content processed by the contentprocessing unit 201 on the basis of the predetermined past videoextracted from the space information stored in the content storage unit220 or the synchronized content generated by the content synchronizingunit 202. The content storage unit 220 is realized by a ROM storingprograms, operation parameters or the like, that are used for processingof the control unit 200 and a RAM temporarily storing parameters or thelike that are appropriately changed.

The configuration of the server 20 according to the present embodimenthas been specifically described hereinabove. Note that the configurationof the server 20 illustrated in FIG. 17 is an example, and the presentembodiment is not limited thereto. For example, at least some ofconfigurations of the server 20 may be in an external apparatus, or atleast some of respective functions of the control unit 200 may berealized by the information processing apparatus 10 or an informationprocessing terminal (for example, a so-called edge server or the like)of which communication distance is relatively close to the informationprocessing apparatus 10. As such, by appropriately distributing therespective configurations of the server 20, it is possible to improve areal-time property, reduce a processing load, and secure security.

3. OPERATION PROCESSING

Next, operation processing of the information processing systemaccording to the present embodiment will be specifically described withreference to FIGS. 18 and 19. FIG. 18 is a flowchart illustratingoperation processing of the information processing apparatus 10A in acase of executing the recollection function according to the presentembodiment. Here, operation processing of the information processingapparatus 10A that performs input and output of the space information inspace A as an example on the assumption that space A and space B arealways connected to each other will be described, but the informationprocessing apparatus 10B that performs input and output of the spaceinformation in space B can also perform similar operation processing.Further, FIG. 19 is a flowchart illustrating operation processing of theserver 20 in a case of executing the recollection function according tothe present embodiment.

<3-1. Operation Processing of Information Processing Apparatus 10ARegarding Execution of Recollection Function>

As illustrated in FIG. 18, first, the information processing apparatus10A acquires real-time space information (a video, an audio, and sensorinformation) of space A from the input unit 110 (step S103).

Next, the information processing apparatus 10A transmits the real-timespace information of space A to the server 20 and the informationprocessing apparatus 10B (step S106).

Next, the information processing apparatus 10A receives real-time spaceinformation of space B from the information processing apparatus 10B(step S109), and performs control to output the real-time spaceinformation of space B from the display 131A and the speaker in space A(step S112).

The information processing apparatus 10A repeats the processing of stepsS103 to S112 to realize an always-on connection with space B. Note thatsimilar processing is also performed in the information processingapparatus 10B, such that real-time space information of space A isoutput from the display 131B and the speaker of space B.

Next, in a case where there is an operation input of the recollectionfunction from a user side (step S115: Yes), the information processingapparatus 10A transmits designated date and time information to theserver 20 (step S118). That is, in a case where the icon 330 c for therecollection function is selected and a predetermined date and time isdesignated by the user, as described with reference to FIGS. 5 and 6,the designated date and time are transmitted to the server 20 as arequest to execute the recollection function.

Next, the information processing apparatus 10A receives the content ofthe designated date and time from the server 20 (step S122).

On the other hand, in a case where there is no operation input of therecollection function from the user side (step S115: No) and there is anoperation input of the recollection function from the counterpart side(step S125: Yes), the information processing apparatus 10A receives acontent of a date and time designated by user B (a user of the alwaysconnected counterpart space) from the server 20 (step S128).

Here, the content of the designated date and time received from theserver 20 is, for example, the past video of the designated date andtime of the counterpart space, a processed content processed on thebasis of the past video of the designated date and time, or asynchronized content in which the past video of the space of the ownside and the past video of the counterpart space in the same time zoneare synchronized with each other.

Next, the information processing apparatus 10A synthesizes the receivedcontent with real-time space information (current video of space B)continuously received from space B, and displays the synthesized contenton the display 131A (step S130). Examples of synthesizing the contentsare as illustrated in FIGS. 7 to 16. The information processingapparatus 10A may output an audio of the past video together from thespeaker.

<3-2. Operation Processing of Server 20 Regarding Execution ofRecollection Function>

Next, operation processing of the server 20 will be described withreference to FIG. 18.

As illustrated in FIG. 18, first, the server 20 receives real-time spaceinformation (a video, an audio, and sensor information) of each space(space A and space B) from the information processing apparatus 10A andthe information processing apparatus 10B, respectively (step S133).

Next, the server 20 stores each received real-time space information inthe content storage unit 220 (step S136).

Next, in a case where there is a request to execute the recollectionfunction from the information processing apparatus 10A or theinformation processing apparatus 10B (step S139: Yes), the server 20extracts the content (the past video of space A and the past video ofspace B) of the designated date and time from the content storage unit220 (step S142).

Next, in a case where processing of the past video is required (stepS145: Yes), the server 20 processes the content by the contentprocessing unit 201 (step S148). Specifically, for example, in a casewhere a silhouette display is selected by the user as a display methodof the recollection function, the content processing unit 201 performsprocessing for extracting a person from the past video and processing asilhouette of the person.

Next, in a case where it is selected for the user to also see the pastvideo of the own side (step S151: Yes) as a display method of therecollection function, the content synchronizing unit 202 of the server20 generates a content (synchronized content) in which time axes of theextracted past videos of space A and space B are synchronized with eachother (step S154). “In a case where it is selected for the user to alsosee the past video of the own side”, it is assumed that a display methodof simultaneously seeing the past videos in the counterpart space andthe space of the own side is selected, as described with reference toFIGS. 8 to 12, 15, and 16.

Then, the server 20 (simultaneously) transmits the generatedsynchronized content to the information processing apparatus 10A and theinformation processing apparatus 10B (step S157). The informationprocessing apparatus 10A and information processing apparatus 10Bdisplay the received synchronized content (content in which the pastvideo of space A and the past video of space B are synchronized witheach other on the time axis in a designated time zone) together with thereal-time videos of the counterpart spaces (see FIGS. 8 to 12, 15, and16).

On the other hand, in a case where it is not selected for the user toalso see the past video of the own side (step S151: No), the server 20(simultaneously) transmits the extracted past video (or processedcontent) of space A and the extracted past video (or processed content)of space B, respectively, to the information processing apparatus 10Band the information processing apparatus 10A (step S160). Theinformation processing apparatus 10A and the information processingapparatus 10B display the received past videos (or processed contents)together with the real-time videos of the counterpart spaces (see FIGS.7, and 11 to 14).

Therefore, it is possible to look back on the past two-way communicationtogether while maintaining a state where a plurality of spaces areadjacently connected to each other. Further, since the past videos andthe like are simultaneously transmitted from the server 20 to theinformation processing apparatus 10A and the information processingapparatus 10B, playing timings of the past videos in each informationprocessing apparatus 10 are synchronized with each other, and at thesame time, the past videos and the like of the same time zone can beplayed back, respectively.

4. ANOTHER SYSTEM CONFIGURATION EXAMPLE

The configuration and the operation processing of the informationprocessing system according to the embodiment of the present disclosurehave been described hereinabove, but the configuration of theinformation processing system according to the present embodiment is notlimited to the configuration illustrated in FIG. 3, that is, aserver-client type system configuration including the server 20, theinformation processing apparatus 10A, and the information processingapparatus 10B. For example, the information processing system may berealized by a configuration that does not include the server 20,specifically, a system configuration including an information processingapparatus 10-nA and an information processing apparatus 10-nB asillustrated in FIG. 20. A configuration example and operation processingof information processing apparatuses 10-n (10-1 to 10-4) in a case ofsuch an information processing system 1 x will be specifically describedbelow using a plurality of embodiments.

4-1. First Embodiment

(Configuration)

FIG. 21 is a diagram illustrating an example of a configuration of aninformation processing apparatus 10-1 according to a first embodiment ofanother system configuration example. Note that in the first embodiment,the information processing apparatuses 10-1 in each space have the sameconfiguration. That is, an information processing system 1-1 accordingto the first embodiment includes information processing apparatuses10-1A and 10-1B. In the following description, the informationprocessing apparatus 10-1B is simply referred to as an informationprocessing apparatus 10B.

As illustrated in FIG. 21, the information processing apparatus 10-1includes a control unit 100-1, an input unit 110, an operation I/F 120,an output unit 130, a communication unit 140, and a storage unit 150-1.Note that the same components as those of the configuration of theinformation processing apparatus 10 illustrated in FIG. 3 are denoted bythe same reference numerals and a detailed description thereof areomitted. Further, here, portions different from those of theconfiguration described with reference to FIG. 3 will be mainlydescribed.

The control unit 100-1 according to the present embodiment performscontrol to transmit real-time space information (a video, an audio, andsensor information) of a space of an own side obtained from the inputunit 110 to the information processing apparatus 10B in an alwaysconnected counterpart space, and performs control to accumulate thetransmitted space information in a transmitted content storage unit 151.Further, the control unit 100-1 performs control to accumulate real-timespace information of the counterpart space received from the informationprocessing apparatus 10B in a received content storage unit 152, andperforms control to output the real-time space information from theoutput unit 130. Further, in a case where there is a recollectionfunction execution instruction, the control unit 100-1 extracts(retrieves) a content (that is, the past video, the past audio and thelike) of a designated date and time from the storage unit 150-1.Further, the control unit 100-1 also functions as a function selectingunit 101, an image synthesizing unit 102, an output control unit 103, acontent processing unit 104, and a content synchronizing unit 105, asillustrated in FIG. 21.

The content processing unit 104 has a function of generating a personimage by extracting only a person from a predetermined extracted pastvideo or generating a 3D model of a person appearing in the past videofrom three-dimensional information based on the past video andcorresponding sensor information (sensor information of the same timezone), in the use of a recollection function, similar to the contentprocessing unit 201 described with reference to FIG. 17.

The content synchronizing unit 105 generates a content (also referred toas a synchronized content) in which the extracted predetermined pastvideos of space A and space B are associated (synchronized) with eachother on the same time axis, in a case of displaying the past videos ofboth sides in each space, similar to the content synchronizing unit 202described with reference to FIG. 17.

The storage unit 150-1 includes the transmitted content storage unit 151and the received content storage unit 152. In the transmitted contentstorage unit 151, a content transmitted to another informationprocessing apparatus 10, specifically, space information obtained fromthe input unit 110 of the space of the own side, a processed contentprocessed by the content processing unit 104, or a synchronized contentsynchronized by the content synchronizing unit 105 can be stored. In thereceived content storage unit 152, a content received from anotherinformation processing apparatus 10, specifically, real-time spaceinformation of the counterpart space, a processed content, or asynchronized content can be stored.

(Operation Processing)

FIGS. 22 and 23 are flowcharts illustrating operation processing of theinformation processing apparatus 10-1A according to the first embodimentof another system configuration example.

As illustrated in FIG. 22, first, the information processing apparatus10-1A acquires real-time space information (a video, an audio, andsensor information) of space A (step S203), transmits the real-timespace information of space A to the information processing apparatus 10Bin space B that is an always connected adjacent space (step S206), andstores the real-time space information transmitted to the informationprocessing apparatus 10B in the transmitted content storage unit 151(step S209).

Further, the information processing apparatus 10-1A receives real-timespace information (a video, an audio, and sensor information) of space Bfrom the information processing apparatus 10B (step S212), and storesthe real-time space information of space B in the received contentstorage unit 152 (step S215).

Next, the information processing apparatus 10-1A outputs the real-timespace information of space B from a display 131A and a speaker of spaceA (step S218).

The information processing apparatus 10-1A repeats the processing ofsteps S203 to S218 to realize an always-on connection with space B.Further, similar processing is also performed in the informationprocessing apparatus 10B, such that real-time space information of spaceA is output from a display 131B and a speaker of space B.

As such, in the information processing apparatus 10-1 according to thefirst embodiment, the content transmitted to the counterpart space, thatis, the space information of the space of the own side, and the contentreceived from the counterpart space, that is, the space information ofthe counterpart space are accumulated in the storage unit 150-1.

Next, in a case where there is an operation input of the recollectionfunction from a user side (step S221: Yes), the information processingapparatus 10-1A transmits designated date and time information to theinformation processing apparatus 10B (step S224).

Next, the information processing apparatus 10-1A extracts a content(past video) of the designated date and time from the storage unit 150-1(step S227).

On the other hand, in a case where there is no operation input of therecollection function from the user side (step S221: No) and there is anoperation input of the recollection function from the counterpart side(step S230: Yes), the information processing apparatus 10-1A extracts acontent (past video) of a date and time designated by user B (a user ofthe always connected counterpart space) from the storage unit 150-1(step S233).

Here, the content of the designated date and time extracted from thestorage unit 150-1 is the past video of the counterpart space stored inthe received content storage unit 152, or is the past video of the spaceof the own side stored in the transmitted content storage unit 151 ifnecessary.

Next, in a case where processing of the past video is required (stepS236: Yes), the information processing apparatus 10-1A processes thecontent by the content processing unit 104 (step S239). Specifically,for example, in a case where a silhouette display is selected by theuser as a display method of the recollection function, the contentprocessing unit 104 performs processing for extracting a person from thepast video and processing a silhouette of the person.

Next, in a case where it is selected for the user to also see the pastvideo of the own side (step S242: Yes) as a display method of therecollection function, the information processing apparatus 10-1Agenerates a content (synchronized content) in which time axes of theextracted past videos of space A (own side) and space B (counterpartside) are synchronized with each other by the content synchronizing unit105 (step S245).

Next, the information processing apparatus 10-1A notifies theinformation processing apparatus 10B that synchronization processing hasbeen completed (step S248).

Next, the information processing apparatus 10-1A confirms whether or notthe synchronization processing has also been completed in theinformation processing apparatus 10B (step S257). In the presentembodiment, the information processing apparatus 10B also has a similarconfiguration, and performs operation processing similar to theprocessing illustrated in FIGS. 22 and 23. Therefore, when it isselected for the user to also see the past video of the own side, thesynchronization processing is also performed in the informationprocessing apparatus 10B. Then, in a case where the synchronizationprocessing has been completed in the information processing apparatus10B, a notification indicating that the synchronization processing hasbeen completed is transmitted from the information processing apparatus10B. Therefore, the information processing apparatus 10B can alsoconfirm whether or not the synchronization processing has beencompleted.

On the other hand, in a case where it is not selected for the user toalso see the past video of the own side (step S242: No), the informationprocessing apparatus 10-1A transmits a preparation completion flag tothe information processing apparatus 10B (step S254).

Next, the information processing apparatus 10-1A confirms whether or notthe preparation completion flag has been prepared (step S257). Asdescribed above, the information processing apparatus 10B also performsthe operation processing similar to the processing illustrated in FIGS.22 and 23, and thus, a preparation completion flag is also transmittedfrom the information processing apparatus 10B. The informationprocessing apparatus 10-1A transmits the preparation completion flag tothe information processing apparatus 10B, and determines that thepreparation completion flags has been prepared in a case where thepreparation completion flag is also transmitted from the informationprocessing apparatus 10B.

Then, in a case where the synchronization processing has also beencompleted in the information processing apparatus 10B (step S251: Yes)or in a case where the preparation completion flag has been prepared(step S257: Yes), the information processing apparatus 10-1A displaysthe extracted past video (or the processed content or the synchronizedcontent) of the counterpart space (space B) together with the real-timevideo of the counterpart space (or in a state of synthesizing theextracted past video (or the processed content or the synchronizedcontent) of the counterpart space (space B) with the real-time video ofthe counterpart space) (step S260).

Therefore, it is possible to look back on the past two-way communicationtogether while maintaining a state where the counterpart spaces areadjacently connected to each other. Further, since the display of thepast videos and the like is performed at a timing when both of theinformation processing apparatus 10-1A and the information processingapparatus 10-1B have completed preparation (or have completed thesynchronization processing), playing timings of the past videos in eachinformation processing apparatus 10-1 are synchronized with each other,and at the same time, the past videos and the like of the same time zonecan be played back, respectively.

4-2. Second Embodiment

(Configuration)

FIG. 24 is a diagram illustrating an example of a configuration of aninformation processing apparatus 10-2 according to a first embodiment ofanother system configuration example. Note that in the secondembodiment, the information processing apparatuses 10-2 in each spacehave the same configuration. That is, an information processing system1-2 according to the second embodiment includes information processingapparatuses 10-2A and 10-2B. In the following description, theinformation processing apparatus 10-2B is simply referred to as aninformation processing apparatus 10B.

As illustrated in FIG. 24, the information processing apparatus 10-2includes a control unit 100-2, an input unit 110, an operation I/F 120,an output unit 130, a communication unit 140, and a storage unit 150-2.Note that the same components as those of the configuration of theinformation processing apparatus 10 illustrated in FIG. 3 are denoted bythe same reference numerals and a detailed description thereof areomitted. Further, here, portions different from those of theconfiguration described with reference to FIG. 3 will be mainlydescribed.

The control unit 100-2 according to the present embodiment performscontrol to transmit real-time space information (a video, an audio, andsensor information) of a space of an own side obtained from the inputunit 110 to the information processing apparatus 10B in an alwaysconnected counterpart space. Further, the control unit 100-2 performscontrol to accumulate real-time space information of the counterpartspace received from the information processing apparatus 10B in areceived content storage unit 152, and performs control to output thereal-time space information from the output unit 130. Further, in a casewhere there is a recollection function execution instruction, thecontrol unit 100-2 extracts (retrieves) a content (that is, the pastvideo, the past audio and the like) of a designated date and time fromthe received content storage unit 152.

Further, the control unit 100-2 also functions as a function selectingunit 101, an image synthesizing unit 102, an output control unit 103, acontent processing unit 104, and a content synchronizing unit 105, asillustrated in FIG. 24. The content processing unit 104 and the contentsynchronizing unit 105 are as described with reference to FIG. 21.

The storage unit 150-2 includes the received content storage unit 152.In the received content storage unit 152, similar to FIG. 21, a contentreceived from another information processing apparatus 10, specifically,real-time space information of the counterpart space, a processedcontent, or a synchronized content can be stored.

(Operation Processing)

FIGS. 25 and 26 are flowcharts illustrating operation processing of theinformation processing apparatus 10-2A according to the secondembodiment of another system configuration example.

As illustrated in FIG. 25, first, the information processing apparatus10-2A acquires real-time space information (a video, an audio, andsensor information) of space A (step S303), and transmits the real-timespace information of space A to the information processing apparatus 10Bin space B that is an always connected adjacent space (step S306).

Further, the information processing apparatus 10-2A receives real-timespace information (a video, an audio, and sensor information) of space Bfrom the information processing apparatus 10B (step S309), and storesthe real-time space information of space B in the received contentstorage unit 152 (step S312).

Next, the information processing apparatus 10-2A outputs the real-timespace information of space B from a display 131A and a speaker of spaceA (step S315).

The information processing apparatus 10-2A repeats the processing ofsteps S303 to S315 to realize an always-on connection with space B.Further, similar processing is also performed in the informationprocessing apparatus 10B, such that real-time space information of spaceA is output from a display 131B and a speaker of space B.

As such, in the information processing apparatus 10-2 according to thesecond embodiment, the content transmitted to the counterpart space,that is, the space information of the space of the own side is notstored, and the content received from the counterpart space, that is,the space information of the counterpart space is accumulated in thestorage unit 150-2.

Next, in a case where there is an operation input of the recollectionfunction from a user side (step S318: Yes), the information processingapparatus 10-2A transmits designated date and time information to theinformation processing apparatus 10B (step S321).

Next, the information processing apparatus 10-2A extracts a content (thepast video of the counterpart space) of the designated date and timefrom the received content storage unit 152 (step S324).

On the other hand, in a case where there is no operation input of therecollection function from the user side (step S318: No) and there is anoperation input of the recollection function from the counterpart side(step S327: Yes), the information processing apparatus 10-2A extracts acontent (the past video of the counterpart space) of a date and timedesignated by user B (a user of the always connected counterpart space)from the received content storage unit 152 (step S330).

Next, if necessary, a copy of the extracted content (the past video ofthe counterpart space) is transmitted to the information processingapparatus 10B (step S333). In a phrase “if necessary”, specifically, itis assumed that in a display method of the recollection function, it isselected to also see the past video of the own side. In this case, asdescribed with reference to FIGS. 8 to 12, 15, and 16, the past videosof both of the counterpart space and the space of the own side arerequired, but in the present embodiment, only the past video of thecounterpart space is stored, and the past video of the space of the ownside, that is, the past video of space B is thus required in theinformation processing apparatus 10B. Therefore, the informationprocessing apparatus 10-1A performs control to transmit a copy of thepast video of the counterpart space (that is, the past video of space B)to the information processing apparatus 10B. Since such control issimilarly performed also in the information processing apparatus 10B,the information processing apparatus 10-1A can receive the past video ofthe space of the own side, that is, the past video of space A, from theinformation processing apparatus 10B.

Next, in a case where processing of the past video is required (stepS336: Yes), the information processing apparatus 10-2A processes thecontent by the content processing unit 104 (step S339). Specifically,for example, in a case where a silhouette display is selected by theuser as a display method of the recollection function, the contentprocessing unit 104 performs processing for extracting a person from thepast video and processing a silhouette of the person.

Next, in a case where it is selected for the user to also see the pastvideo of the own side (step S342: Yes) as a display method of therecollection function, the information processing apparatus 10-2Agenerates a content (synchronized content) in which time axes of thereceived past video of space A (own side) and the extracted past videoof space B (counterpart side) are synchronized with each other by thecontent synchronizing unit 105 (step S345).

Next, the information processing apparatus 10-2A notifies theinformation processing apparatus 10B that synchronization processing hasbeen completed (step S348).

Next, the information processing apparatus 10-2A confirms whether or notthe synchronization processing has also been completed in theinformation processing apparatus 10B (step S351).

On the other hand, in a case where it is not selected for the user toalso see the past video of the own side (step S342: No), the informationprocessing apparatus 10-2A transmits a preparation completion flag tothe information processing apparatus 10B (step S354).

Next, the information processing apparatus 10-2A confirms whether or notthe preparation completion flag has been prepared (step S357).

Then, in a case where the synchronization processing has also beencompleted in the information processing apparatus 10B (step S351: Yes)or in a case where the preparation completion flag has been prepared(step S357: Yes), the information processing apparatus 10-2A displaysthe extracted past video (or the processed content or the synchronizedcontent) of the counterpart space (space B) together with the real-timevideo of the counterpart space (or in a state of synthesizing theextracted past video (or the processed content or the synchronizedcontent) of the counterpart space (space B) with the real-time video ofthe counterpart space) (step S360).

Therefore, it is possible to look back on the past two-way communicationtogether while maintaining a state where the counterpart spaces areadjacently connected to each other. Further, since the display of thepast videos and the like is performed at a timing when both of theinformation processing apparatus 10-2A and the information processingapparatus 10-2B have completed preparation (or have completed thesynchronization processing), playing timings of the past videos in eachinformation processing apparatus 10-2 are synchronized with each other,and at the same time, the past videos and the like of the same time zonecan be played back, respectively.

4-3. Third Embodiment

(Configuration)

FIG. 27 is a diagram illustrating an example of a configuration of aninformation processing apparatus 10-3 according to a third embodiment ofanother system configuration example. Note that in the third embodiment,the information processing apparatuses 10-3 in each space have the sameconfiguration. That is, an information processing system 1-3 accordingto the third embodiment includes information processing apparatuses10-3A and 10-3B. In the following description, the informationprocessing apparatus 10-3B is simply referred to as an informationprocessing apparatus 10B.

As illustrated in FIG. 27, the information processing apparatus 10-3includes a control unit 100-3, an input unit 110, an operation I/F 120,an output unit 130, a communication unit 140, and a storage unit 150-3.Note that the same components as those of the configuration of theinformation processing apparatus 10 illustrated in FIG. 3 are denoted bythe same reference numerals and a detailed description thereof areomitted. Further, here, portions different from those of theconfiguration described with reference to FIG. 3 will be mainlydescribed.

The control unit 100-3 according to the present embodiment performscontrol to transmit real-time space information (a video, an audio, andsensor information) of a space of an own side obtained from the inputunit 110 to the information processing apparatus 10B in an alwaysconnected counterpart space, and performs control to accumulate thetransmitted space information in a transmitted content storage unit 151.Further, the control unit 100-3 performs control to output real-timespace information of the counterpart space received from the informationprocessing apparatus 10B from the output unit 130. Further, in a casewhere there is a recollection function execution instruction, thecontrol unit 100-2 extracts (retrieves) a content (that is, the pastvideo, the past audio and the like) of a designated date and time fromthe transmitted content storage unit 151.

Further, the control unit 100-3 also functions as a function selectingunit 101, an image synthesizing unit 102, an output control unit 103, acontent processing unit 104, and a content synchronizing unit 105, asillustrated in FIG. 27. The content processing unit 104 and the contentsynchronizing unit 105 are as described with reference to FIG. 21.

The storage unit 150-3 includes the transmitted content storage unit151. In the transmitted content storage unit 151, a content transmittedto another information processing apparatus 10, specifically, spaceinformation obtained from the input unit 110 of the space of the ownside, a processed content processed by the content processing unit 104,or a synchronized content synchronized by the content synchronizing unit105 can be stored.

(Operation Processing)

FIGS. 28 and 29 are flowcharts illustrating operation processing of theinformation processing apparatus 10-3A according to the third embodimentof another system configuration example.

As illustrated in FIG. 28, first, the information processing apparatus10-3A acquires real-time space information (a video, an audio, andsensor information) of space A (step S403), transmits the real-timespace information of space A to the information processing apparatus 10Bin space B that is an always connected adjacent space (step S406), andstores the real-time space information transmitted to the informationprocessing apparatus 10B in the transmitted content storage unit 151(step S409).

Further, the information processing apparatus 10-3A receives real-timespace information (a video, an audio, and sensor information) of space Bfrom the information processing apparatus 10B (step S412), and outputsthe real-time space information of space B from a display 131A and aspeaker of space A (step S415).

The information processing apparatus 10-1A repeats the processing ofsteps S403 to S415 to realize an always-on connection with space B.Further, similar processing is also performed in the informationprocessing apparatus 10B, such that real-time space information of spaceA is output from a display 131B and a speaker of space B.

As such, in the information processing apparatus 10-3 according to thethird embodiment, the content transmitted to the counterpart space, thatis, the space information of the space of the own side is stored in thetransmitted content storage unit 151, and the content received from thecounterpart space is not stored.

Next, in a case where there is an operation input of the recollectionfunction from a user side (step S418: Yes), the information processingapparatus 10-3A transmits designated date and time information to theinformation processing apparatus 10B (step S421).

Next, the information processing apparatus 10-3A extracts a content (thepast video of the space of the own side) of the designated date and timefrom the transmitted content storage unit 151 (step S424).

On the other hand, in a case where there is no operation input of therecollection function from the user side (step S418: No) and there is anoperation input of the recollection function from the counterpart side(step S427: Yes), the information processing apparatus 10-3A extracts acontent (the past video of the space of the own side) of a date and timedesignated by user B (a user of the always connected counterpart space)from the transmitted content storage unit 151 (step S430).

Next, the extracted content (the past video of the space of the ownside) is transmitted to the information processing apparatus 10B (stepS433). At least the past video of the counterpart space is required todisplay the recollection function, but in the present embodiment, onlythe past video of the space of the own side is stored, and the pastvideo of the counterpart space, that is, the past video of space A isthus required in the information processing apparatus 10B. Therefore,the information processing apparatus 10-3A performs control to transmitthe past video of the space of the own side (that is, the past video ofspace A) to the information processing apparatus 10B. Since such controlis similarly performed also in the information processing apparatus 10B,the information processing apparatus 10-3A can receive the past video ofthe counterpart space, that is, the past video of space B, from theinformation processing apparatus 10B.

Next, in a case where processing of the past video is required (stepS436: Yes), the information processing apparatus 10-3A processes thecontent by the content processing unit 104 (step S439).

Next, in a case where it is selected for the user to also see the pastvideo of the own side (step S442: Yes) as a display method of therecollection function, the information processing apparatus 10-3Agenerates a content (synchronized content) in which time axes of theextracted past video of space A (own side) and the received past videoof space B (counterpart side) are synchronized with each other by thecontent synchronizing unit 105 (step S445).

Next, the information processing apparatus 10-3A notifies theinformation processing apparatus 10B that synchronization processing hasbeen completed (step S448).

Next, the information processing apparatus 10-3A confirms whether or notthe synchronization processing has also been completed in theinformation processing apparatus 10B (step S451).

On the other hand, in a case where it is not selected for the user toalso see the past video of the own side (step S442: No), the informationprocessing apparatus 10-3A transmits a preparation completion flag tothe information processing apparatus 10B (step S454).

Next, the information processing apparatus 10-3A confirms whether or notthe preparation completion flag has been prepared (step S457).

Then, in a case where the synchronization processing has also beencompleted in the information processing apparatus 10B (step S451: Yes)or in a case where the preparation completion flag has been prepared(step S457: Yes), the information processing apparatus 10-3A displaysthe received past video (or the processed content or the synchronizedcontent) of the counterpart space (space B) together with the real-timevideo of the counterpart space (or in a state of synthesizing thereceived past video (or the processed content or the synchronizedcontent) of the counterpart space (space B) with the real-time video ofthe counterpart space) (step S460).

Therefore, it is possible to look back on the past two-way communicationtogether while maintaining a state where the counterpart spaces areadjacently connected to each other. Further, since the display of thepast videos and the like is performed at a timing when both of theinformation processing apparatus 10-3A and the information processingapparatus 10-3B have completed preparation (or have completed thesynchronization processing), playing timings of the past videos in eachinformation processing apparatus 10-3 are synchronized with each other,and at the same time, the past videos and the like of the same time zonecan be played back, respectively.

4-4. Fourth Embodiment

(Configuration)

FIG. 30 is a diagram illustrating an example of a configuration of aninformation processing apparatus 10-4 according to a fourth embodimentof another system configuration example. An information processingsystem 1-4 according to the fourth embodiment includes informationprocessing apparatuses 10-4A and 10-4B. Here, one information processingapparatus 10-4 includes a “content processing unit” and a “contentsynchronizing unit”.

As illustrated in FIG. 30, the information processing apparatus 10-4Aincludes a control unit 100-4A, an input unit 110, an operation I/F 120,an output unit 130, a communication unit 140, and a storage unit 150-4A,and the control unit 100-4A also functions as a content processing unit104 and a content synchronizing unit 105.

On the other hand, the information processing apparatus 10-4B includes acontrol unit 100-4B, an input unit 110, an operation I/F 120, an outputunit 130, a communication unit 140, and a storage unit 150-4B, and thecontrol unit 100-4B does not have a content processing function or acontent synchronizing function.

Operation processing of the information processing system in a casewhere one information processing apparatus 10-4A has the contentprocessing function or the content synchronizing function as describedabove will be described below with reference to FIGS. 31 to 34. FIGS. 31and 32 are flowcharts illustrating operation processing of theinformation processing apparatus 10-4A having the content processingfunction or the content synchronizing function. Further, FIGS. 33 and 34are flowcharts illustrating operation processing of the informationprocessing apparatus 10-4B that does not have the content processingfunction or the content synchronizing function. In the followingdescription, the information processing apparatus 10-4B is simplyreferred to as an information processing apparatus 10B.

(Operation Processing of Information Processing Apparatus 10-4A)

As illustrated in FIG. 31, first, the information processing apparatus10-4A acquires real-time space information (a video, an audio, andsensor information) of space A (step S503), transmits the real-timespace information of space A to the information processing apparatus 10Bin space B that is an always connected adjacent space (step S506), andstores the real-time space information transmitted to the informationprocessing apparatus 10B in the transmitted content storage unit 151(step S509).

Further, the information processing apparatus 10-4A receives real-timespace information (a video, an audio, and sensor information) of space Bfrom the information processing apparatus 10B (step S512), and outputsthe real-time space information of space B from a display 131A and aspeaker of space A (step S515).

The information processing apparatus 10-4A repeats the processing ofsteps S503 to S515 to realize an always-on connection with space B.Further, similar processing is also performed in the informationprocessing apparatus 10B, such that real-time space information of spaceA is output from a display 131B and a speaker of space B.

As such, in the information processing apparatus 10-4A according to thefourth embodiment, the content transmitted to the counterpart space,that is, the space information of the space of the own side is stored inthe transmitted content storage unit 151, and the content received fromthe counterpart space is not stored.

Next, in a case where there is an operation input of the recollectionfunction from a user side (step S518: Yes), the information processingapparatus 10-4A transmits designated date and time information to theinformation processing apparatus 10B (step S521).

Next, the information processing apparatus 10-4A extracts a content (thepast video of the space of the own side) of the designated date and timefrom the transmitted content storage unit 151 (step S524).

On the other hand, in a case where there is no operation input of therecollection function from the user side (step S518: No) and there is anoperation input of the recollection function from the counterpart side(step S527: Yes), the information processing apparatus 10-4A extracts acontent (the past video of the space of the own side) of a date and timedesignated by user B (a user of the always connected counterpart space)from the transmitted content storage unit 151 (step S530).

Next, the extracted content (the past video of the space of the ownside) is transmitted to the information processing apparatus 10B (stepS533). Details are similar to those of step S433 described in the thirdembodiment.

Next, in a case where processing of the past video is required (stepS536: Yes), the information processing apparatus 10-4A processes thecontent by the content processing unit 104 (step S539).

Next, in a case where it is selected for the user to also see the pastvideo of the own side (step S542: Yes) as a display method of therecollection function, the information processing apparatus 10-4Agenerates a content (synchronized content) in which time axes of theextracted past video of space A (own side) and the received past videoof space B (counterpart side) are synchronized with each other by thecontent synchronizing unit 105 (step S545).

Next, the information processing apparatus 10-4A transmits a copy of thesynchronized content to the information processing apparatus 10B (stepS548). In the present embodiment, since the information processingapparatus 10B does not have the content synchronizing function, thesynchronized content generated by the information processing apparatus10A is transmitted.

On the other hand, in a case where it is not selected for the user toalso see the past video of the own side (step S542: No), the informationprocessing apparatus 10-4A transmits the processed content to theinformation processing apparatus 10B (step S551) in a case where theprocessing is performed. In the present embodiment, since theinformation processing apparatus 10B does not have the contentprocessing function, the processed content generated by the informationprocessing apparatus 10A is transmitted. Note that in a case where theprocessing is not performed, similar to the third embodiment, apreparation completion flag may be transmitted and it may be confirmedwhether or not the preparation completion flag has been prepared.

Then, the information processing apparatus 10-4A displays the receivedpast video (or the processed content or the synchronized content) of thecounterpart space (space B) together with the real-time video of thecounterpart space (or in a state of synthesizing the received past video(or the processed content or the synchronized content) of thecounterpart space (space B) with the real-time video of the counterpartspace) (step S554).

(Operation Processing of Information Processing Apparatus 10-4B)

Next, operation processing of the information processing apparatus 10-4Bthat does not have the content processing function or the contentsynchronizing function will be described with reference to FIGS. 33 and34. Here, the information processing apparatus 10-4A is simply referredto as an information processing apparatus 10A.

As illustrated in FIG. 33, first, the information processing apparatus10-4B acquires real-time space information (a video, an audio, andsensor information) of space B (step S573), transmits the real-timespace information of space B to the information processing apparatus 10Ain space A that is an always connected adjacent space (step S576), andstores the real-time space information transmitted to the informationprocessing apparatus 10A in the transmitted content storage unit 151(step S579).

Further, the information processing apparatus 10-4B receives real-timespace information (a video, an audio, and sensor information) of space Afrom the information processing apparatus 10A (step S582), and outputsthe real-time space information of space A from a display 131B and aspeaker of space B (step S585).

The information processing apparatus 10-4B repeats the processing ofsteps S573 to S585 to realize an always-on connection with space A.

As such, in the information processing apparatus 10-4B according to thefourth embodiment, the content transmitted to the counterpart space,that is, the space information of the space of the own side is stored inthe transmitted content storage unit 151, and the content received fromthe counterpart space is not stored.

Next, in a case where there is an operation input of the recollectionfunction from a user side (step S588: Yes), the information processingapparatus 10-4B transmits designated date and time information to theinformation processing apparatus 10A (step S591).

Next, the information processing apparatus 10-4B extracts a content (thepast video of the space of the own side) of the designated date and timefrom the transmitted content storage unit 151 (step S594).

On the other hand, in a case where there is no operation input of therecollection function from the user side (step S588: No) and there is anoperation input of the recollection function from the counterpart side(step S597: Yes), the information processing apparatus 10-4B extracts acontent (the past video of the space of the own side) of a date and timedesignated by user A (a user of the always connected counterpart space)from the transmitted content storage unit 151 (step S600).

Next, the extracted content (the past video of the space of the ownside) is transmitted to the information processing apparatus 10A (stepS603). Details are similar to those of step S433 described in the fourthembodiment.

Next, in a case where it is selected for the user to also see the pastvideo of the own side (step S606: Yes) as a display method of therecollection function, the information processing apparatus 10-4Breceives the synchronized content from the information processingapparatus 10A (step S609).

On the other hand, in a case where it is not selected for the user toalso see the past video of the own side (step S606: No), the informationprocessing apparatus 10-4B receives the processed content from theinformation processing apparatus 10A (step S612) in a case whereprocessing of the past video is required. Note that in a case where theprocessing is not required, similar to the third embodiment, apreparation completion flag may be transmitted and it may be confirmedwhether or not the preparation completion flag has been prepared.

In the present embodiment, since the information processing apparatus10B does not have the content synchronizing function or the contentprocessing function, in a case where of processing or synchronization isrequired in a display method of the memory function, the informationprocessing apparatus 10B receives the processed content or thesynchronized content generated by the information processing apparatus10A.

Then, the information processing apparatus 10-4B displays the receivedpast video (or the processed content or the synchronized content) of thecounterpart space (space A) together with the real-time video of thecounterpart space (or in a state of synthesizing the received past video(or the processed content or the synchronized content) of thecounterpart space (space A) with the real-time video of the counterpartspace) (step S615).

As described above, also in the fourth embodiment, it is possible tolook back on the past two-way communication together while maintaining astate where the counterpart spaces are adjacently connected to eachother. Further, since the display of the past videos and the like isperformed at a timing when the synchronized content or the processedcontent is transmitted from the information processing apparatus 10-4Ato the information processing apparatus 10-4B or a timing when both ofthe information processing apparatus 10-4A and the informationprocessing apparatus 10-4B have completed preparation, playing timingsof the past videos in each information processing apparatus 10-4 aresynchronized with each other, and at the same time, the past videos andthe like of the same time zone can be played back, respectively.

5. HARDWARE CONFIGURATION

Finally, a hardware configuration of the information processingapparatus according to the present embodiment will be described withreference to FIG. 35. FIG. 35 is a block diagram illustrating an exampleof a hardware configuration of the information processing apparatusaccording to the present embodiment. Note that the informationprocessing apparatus 900 illustrated in FIG. 35 can realize, forexample, the information processing apparatus 10 illustrated in FIG. 4,the information processing apparatus 10-1 illustrated in FIG. 21, theinformation processing apparatus 10-2 illustrated in FIG. 24, theinformation processing apparatus 10-3 illustrated in FIG. 27, theinformation processing apparatus 10-4 (10-4A and 10-4B) illustrated inFIG. 30, or the server 20 illustrated in FIG. 17. Information processingby the information processing apparatuses 10 and 10-1 to 10-4 or theserver 20 according to the present embodiment is realized by cooperationbetween software and hardware as described later.

As illustrated in FIG. 9, the information processing apparatus 900includes a CPU 901, a ROM 902, a RAM 903, and a host bus 904 a. Further,the information processing apparatus 900 includes a bridge 904, anexternal bus 904 b, an interface 905, an input device 906, an outputdevice 907, a storage device 908, a drive 909, a connection port 911,and a communication device 913. The information processing apparatus 900may have a processing circuit such as an electric circuit, a digitalsignal processor (DSP), an application specific integrated chip (ASIC),or the like, instead of or together with the CPU 901.

The CPU 901 functions as an arithmetic processing device and a controldevice, and generally controls an operation in the informationprocessing apparatus 900 according to various programs. Further, the CPU901 may be a microprocessor. The ROM 902 stores programs, operationparameters, or the like used by the CPU 901. The RAM 903 temporarilystores programs used in execution of the CPU 901, parametersappropriately changed in the execution, or the like. The CPU 901 canform, for example, the control unit 100 illustrated in FIG. 4, thecontrol unit 200 illustrated in FIG. 17, the control unit 100-1illustrated in FIG. 21, the control unit 100-2 illustrated in FIG. 24,the control unit 100-3 illustrated in FIG. 27, or the control unit 100-4illustrated in FIG. 30.

The CPU 901, the ROM 902, and the RAM 903 are connected to each other bythe host bus 904 a including a CPU bus or the like. The host bus 904 ais connected to the external bus 904 b such as a peripheral componentinterconnect/interface (PCI) bus or the like through the bridge 904.Note that the host bus 904 a, the bridge 904, and the external bus 904 bdo not necessarily need to be separately configured, and functions ofthe host bus 904 a, the bridge 904, and the external bus 904 b may bemounted on a single bus.

The input device 906 is realized by, for example, a device such as amouse, a keyboard, a touch panel, a button, a microphone, a switch, alever, or the like, to which information is input by a user. Further,the input device 906 may be, for example, a remote control device usinginfrared rays or other electric waves, or may be an external connectiondevice such as a mobile phone, a personal digital assistants (PDA) orthe like corresponding to an operation of the information processingapparatus 900. Moreover, the input device 906 may include, for example,an input control circuit or the like that generates an input signal onthe basis of the information input by the user using the input means andoutputs the generated input signal to the CPU 901. The user of theinformation processing apparatus 900 can input various data to theinformation processing apparatus 900 or instruct the informationprocessing apparatus 900 to perform a processing operation by operatingthe input device 906.

The output device 907 is a device that can visually or auditorily notifythe user of the acquired information. Such a device includes a displaydevice such as a cathode ray tube (CRT) display device, a liquid crystaldisplay device, a plasma display device, an electroluminescence (EL)display device, a laser projector, a light emitting diode (LED)projector, a lamp or the like, an audio output device such as a speaker,a headphone or the like, a printer device, or the like. The outputdevice 907 outputs, for example, results obtained by various processingperformed by the information processing apparatus 900. Specifically, thedisplay device visually displays the results obtained by the variousprocessing performed by the information processing apparatus 900 invarious formats such as a text, an image, a table, a graph, and thelike. On the other hand, the audio output device converts an audiosignal including played audio data, acoustic data or the like into ananalog signal and auditorily outputs the analog signal.

The storage device 908 is a device for data storage formed as an exampleof a storage unit of the information processing apparatus 900. Thestorage device 908 is realized by, for example, a magnetic storage unitdevice such as a hard disk drive (HDD), a semiconductor storage device,an optical storage device, a magneto-optical storage device, or thelike. The storage device 908 may include a storage medium, a recordingdevice recording data in the storage medium, a reading device readingdata from the storage medium, a deleting device deleting data recordedin the storage medium, and the like. The storage device 908 storesprograms executed by the CPU 901 or various data, various data acquiredfrom the outside, and the like. The storage device 908 can form, forexample, the storage unit 150 illustrated in FIG. 4, the storage unit220 illustrated in FIG. 17, the storage unit 150-1 illustrated in FIG.21, the storage unit 150-2 illustrated in FIG. 24, the storage unit150-3 illustrated in FIG. 27, or the storage unit 150-4 illustrated inFIG. 30.

The drive 909 is a reader/writer for the storage medium, and is embeddedin or externally mounted on the information processing apparatus 900.The drive 909 reads information recorded in a removable storage mediumsuch as a magnetic disk, an optical disk, a magneto-optical disk, asemiconductor memory, or the like, that is mounted, and outputs the readinformation to the RAM 903. Further, the drive 909 can write informationto the removable storage medium.

The connection port 911 is an interface connected to an external device,and is a connection port to an external device capable of transmittingdata by, for example, a universal serial bus (USB), infrared rays,Bluetooth (registered trademark), Wi-Fi (registered trademark), or thelike. The connection port 911 can form, for example, the operation I/F120 illustrated in FIG. 4.

The communication device 913 is, for example, a communication interfaceincluding a communication device or the like for being connected to anetwork 920. The communication device 913 is, for example, acommunication card or the like for wired or wireless local area network(LAN), long term evolution (LTE), Bluetooth (registered trademark), orwireless USB (WUSB). Further, the communication device 913 may be arouter for optical communication, a router for asymmetric digitalsubscriber line (ADSLmodems for various types of communications, or thelike. The communication device 913 can transmit and receive a signal orthe like, for example, to or from the Internet or another communicationdevice according to a predetermined protocol such as, for example,transmission control protocol/Internet protocol (TCP/IP) or the like.The communication device 913 can form, for example, the communicationunit 140 illustrated in FIG. 4, FIG. 21, FIG. 24, FIG. 27, or FIG. 30,or the communication unit 210 illustrated in FIG. 17.

Note that the network 920 is a wired or wireless transmission path ofinformation transmitted from a device connected to the network 920. Forexample, the network 920 may include a public network such as theInternet, a telephone network, a satellite communication network or thelike, various local area networks (LANs) including Ethernet (registeredtrademark), a wide area network (WAN), or the like. Further, the network920 may include a dedicated line network such as the Internetprotocol-virtual private network (IP-VPN) or the like.

An example of the hardware configuration capable of realizing theinformation processing apparatus 900 according to the present embodimenthas been described hereinabove. Each component described above may berealized using a general-purpose member or may be realized by hardwarespecialized for the function of each component. Therefore, it ispossible to appropriately change the hardware configuration to be usedaccording to a technical level at the time of carrying out the presentembodiment.

Note that a computer program for realizing each function of theinformation processing apparatus 900 according to the present embodimentas described above can be created and mounted in a personal computer(PC) or the like. Further, a computer readable recording medium in whichsuch a computer program is stored can be provided. The computer readablerecording medium is, for example, a magnetic disk, an optical disk, amagneto-optical disk, a flash memory or the like. Further, the computerprogram described above may be distributed through, for example, anetwork without using the computer readable recording medium.

6. SUMMARY

As described above, in the information processing system according tothe embodiment of the present disclosure, it is possible to look back onthe past two-way communication while maintaining connection betweenspaces in real time.

Further, by extracting and playing back only the person from the pastvideo, a difference between the present and the past can be felt, andthe past can thus be recollected more intuitively. Since the past personis superimposed and displayed on a video of a current space, it ispossible to intuitively grasp growth of a child, or the like.

Further, by providing a function of looking back on the past,performance of communication with an always connected counterpart istriggered, such that a communication opportunity is increased.

Hereinabove, the preferred embodiments of the present disclosure havebeen described in detail with reference to the accompanying drawings,but the present technology is not limited to such embodiments. It willbe apparent to those skilled in the art of the present disclosure thatvarious modifications or alterations can be conceived within the scopeof the technical idea described in the claims, and it is naturallyunderstood that these modifications or alterations also fall within thetechnical scope of the present disclosure.

For example, it is also possible to create a computer program forcausing hardware such as the CPU, the ROM, the RAM, and the like,embedded in the information processing apparatus 10 or the server 20described above to exert a function of the information processingapparatus 10 or the server 20. Further, a computer readable recordingmedium in which the computer program is stored is also provided.

Further, the effects described in the present specification are onlyillustrative or exemplary rather than being restrictive. That is, thetechnology according to the present disclosure can accomplish othereffects apparent to those skilled in the art from the description of thepresent specification, in addition to or instead of the effectsdescribed above.

Note that the present technology can also have the followingconfiguration.

(1)

An information processing apparatus that controls two-way communicationbetween a first space and a second space, comprising:

a control unit that performs control to display first image informationand second image information in the first space and performs control todisplay third image information and fourth image information in thesecond space, wherein

the first image information is current image information regarding thesecond space,

the second image information is past image information regarding thesecond space,

the third image information is current image information regarding thefirst space, and

the fourth image information is past image information regarding thefirst space in the same time zone as that of the second imageinformation.

(2)

An information processing apparatus in which control of two-waycommunication between a first space and a second space is performed,comprising:

a control unit that performs control to display first image informationand second image information in the first space, wherein

the first image information is current image information regarding thesecond space, and

the second image information

is past image information regarding the second space and

is image information of the same time zone as that of past imageinformation regarding the first space displayed in the second space.

(3)

The information processing apparatus according to (2), wherein thecontrol unit performs control to display the second image information ona virtual space provided in front of the first image information.

(4)

The information processing apparatus according to (2) or (3), whereinthe control unit performs control to display a content in which the pastimage information regarding the first space and the second imageinformation are synchronized with each other on a time axis, togetherwith the first image information.

(5)

The information processing apparatus according to any one of (2) to (4),wherein the control unit performs control to superimpose and display acontent processed based on the second image information, on the firstimage information.

(6)

The information processing apparatus according to (5), wherein theprocessed content is a processed image of a person extracted from thesecond image information.

(7)

The information processing apparatus according to (6), wherein thecontrol unit controls a position at which the processed image of theextracted person is superimposed and displayed on the first imageinformation based on depth information of the person.

(8)

The information processing apparatus according to (6) or (7), whereinthe processed image of the person is a silhouette image of the extractedperson.

(9)

The information processing apparatus according to (6), wherein theprocessed image of the person is a 3D modeled image of the extractedperson, and is displayed on a virtual space provided in front of thefirst image information in a display unit of the first space.

(10)

An information processing method that controls two-way communicationbetween a first space and a second space, comprising:

performing control to display first image information and second imageinformation in the first space and performing control to display thirdimage information and fourth image information in the second space, by aprocessor, wherein

the first image information is current image information regarding thesecond space,

the second image information is past image information regarding thesecond space,

the third image information is current image information regarding thefirst space, and

the fourth image information is past image information regarding thefirst space in the same time zone as that of the second imageinformation.

(11)

A program that causes a computer to function as an informationprocessing apparatus that controls two-way communication between a firstspace and a second space, wherein

the program causes the computer to further function as a control unitthat performs control to display first image information and secondimage information in the first space and performs control to displaythird image information and fourth image information in the secondspace,

the first image information being current image information regardingthe second space,

the second image information being past image information regarding thesecond space,

the third image information being current image information regardingthe first space, and

the fourth image information being past image information regarding thefirst space in the same time zone as that of the second imageinformation.

(12)

An information processing method in which control of two-waycommunication between a first space and a second space is performed,comprising:

performing control to display first image information and second imageinformation in the first space, by a processor, wherein

the first image information is current image information regarding thesecond space, and

the second image information

is past image information regarding the second space and

is image information of the same time zone as that of past imageinformation regarding the first space displayed in the second space.

(13)

A program that causes a computer to function as an informationprocessing apparatus in which control of two-way communication between afirst space and a second space is performed and

causes the computer to further function as a control unit that performscontrol to display first image information and second image informationin the first space, wherein

the first image information is current image information regarding thesecond space, and

the second image information

is past image information regarding the second space and

is image information of the same time zone as that of past imageinformation regarding the first space displayed in the second space.

REFERENCE SIGNS LIST

-   -   1 INFORMATION PROCESSING SYSTEM    -   10(10A, 10B), 10-1 to 10-4 INFORMATION PROCESSING APPARATUS    -   20 SERVER    -   30 NETWORK    -   100 CONTROL UNIT    -   101 FUNCTION SELECTING UNIT    -   102 IMAGE SYNTHESIZING UNIT    -   103 OUTPUT CONTROL UNIT    -   104 CONTENT PROCESSING UNIT    -   105 CONTENT SYNCHRONIZING UNIT    -   110 INPUT UNIT    -   120 OPERATION I/F    -   130 OUTPUT UNIT    -   131 DISPLAY    -   140 COMMUNICATION UNIT    -   150 STORAGE UNIT    -   151 TRANSMITTED CONTENT STORAGE UNIT    -   152 RECEIVED CONTENT STORAGE UNIT    -   200 CONTROL UNIT    -   201 CONTENT PROCESSING UNIT    -   202 CONTENT SYNCHRONIZING UNIT    -   210 COMMUNICATION UNIT    -   220 CONTENT STORAGE UNIT

1. An information processing apparatus that controls two-waycommunication between a first space and a second space, comprising: acontrol unit that performs control to display first image informationand second image information in the first space and performs control todisplay third image information and fourth image information in thesecond space, wherein the first image information is current imageinformation regarding the second space, the second image information ispast image information regarding the second space, the third imageinformation is current image information regarding the first space, andthe fourth image information is past image information regarding thefirst space in the same time zone as that of the second imageinformation.
 2. An information processing apparatus in which control oftwo-way communication between a first space and a second space isperformed, comprising: a control unit that performs control to displayfirst image information and second image information in the first space,wherein the first image information is current image informationregarding the second space, and the second image information is pastimage information regarding the second space and is image information ofthe same time zone as that of past image information regarding the firstspace displayed in the second space.
 3. The information processingapparatus according to claim 2, wherein the control unit performscontrol to display the second image information on a virtual spaceprovided in front of the first image information.
 4. The informationprocessing apparatus according to claim 2, wherein the control unitperforms control to display a content in which the past imageinformation regarding the first space and the second image informationare synchronized with each other on a time axis, together with the firstimage information.
 5. The information processing apparatus according toclaim 2, wherein the control unit performs control to superimpose anddisplay a content processed based on the second image information, onthe first image information.
 6. The information processing apparatusaccording to claim 5, wherein the processed content is a processed imageof a person extracted from the second image information.
 7. Theinformation processing apparatus according to claim 6, wherein thecontrol unit controls a position at which the processed image of theextracted person is superimposed and displayed on the first imageinformation based on depth information of the person.
 8. The informationprocessing apparatus according to claim 6, wherein the processed imageof the person is a silhouette image of the extracted person.
 9. Theinformation processing apparatus according to claim 6, wherein theprocessed image of the person is a 3D modeled image of the extractedperson, and is displayed on a virtual space provided in front of thefirst image information in a display unit of the first space.
 10. Aninformation processing method that controls two-way communicationbetween a first space and a second space, comprising: performing controlto display first image information and second image information in thefirst space and performing control to display third image informationand fourth image information in the second space, by a processor,wherein the first image information is current image informationregarding the second space, the second image information is past imageinformation regarding the second space, the third image information iscurrent image information regarding the first space, and the fourthimage information is past image information regarding the first space inthe same time zone as that of the second image information.
 11. Aprogram that causes a computer to function as an information processingapparatus that controls two-way communication between a first space anda second space, wherein the program causes the computer to furtherfunction as a control unit that performs control to display first imageinformation and second image information in the first space and performscontrol to display third image information and fourth image informationin the second space, the first image information being current imageinformation regarding the second space, the second image informationbeing past image information regarding the second space, the third imageinformation being current image information regarding the first space,and the fourth image information being past image information regardingthe first space in the same time zone as that of the second imageinformation.
 12. An information processing method in which control oftwo-way communication between a first space and a second space isperformed, comprising: performing control to display first imageinformation and second image information in the first space, by aprocessor, wherein the first image information is current imageinformation regarding the second space, and the second image informationis past image information regarding the second space and is imageinformation of the same time zone as that of past image informationregarding the first space displayed in the second space.
 13. A programthat causes a computer to function as an information processingapparatus in which control of two-way communication between a firstspace and a second space is performed and causes the computer to furtherfunction as a control unit that performs control to display first imageinformation and second image information in the first space, wherein thefirst image information is current image information regarding thesecond space, and the second image information is past image informationregarding the second space and is image information of the same timezone as that of past image information regarding the first spacedisplayed in the second space.